I have found that most "highly qualified men" can remain in their positions as long as they have the support of the elder body they serve with. I recall a CO that traveled through a congregation I was in and he apparently decided that one of the elders simply did not qualify because his oldest kid, an adult, had rejected the "truth." I don't believe he knew the circumstances. It appeared that he simply wanted him out. He apparently wrote his letter along with his report and the elder body reportedly wrote one to rebut his position. He remained an elder.
I would imagine that if there was a finger pointer among the group, the elder would have been deleted. But, we all saw the wisdom of the "cast the first stone" portion of the Bible.
SOP